fix(eval): repair broken targets.yaml and two eval-run bugs found during dogfood - #1702

Summary

Found while dogfooding the av-cpl5 artifact-boundary work (previous session):

• .agentv/targets.yaml failed validation on main. Two same-day merges (f0f768f2, ce83a4dd) hard-rejected the legacy ${{ VAR }} interpolation syntax and removed the use_target field with no replacement mechanism, but never migrated this repo-root file. Every agentv eval run using the default targets file was broken.

  • Migrated all scalar fields to {{ env.VAR }} (straight semantic drop-in, confirmed against the framework's own migration codemod).
  • default/agent/llm/grader used use_target to redirect to whatever target name an env var (AGENT_TARGET/LLM_TARGET/GRADER_TARGET) pointed to. Since target aliasing has no replacement, and 65+ example eval files (examples/**/*.eval.yaml) plus 12 target blocks explicitly reference target: llm / grader_target: grader, these are now concrete targets mirroring today's env values (copilot-cli / azure / azure) instead of aliases. This keeps every example file resolving with zero edits to those files. The tradeoff: switching the active provider now means editing targets.yaml directly instead of one env var — a real framework limitation (use_target removal shipped with no migration path), not something fixable from this file alone.
  • Note: I evaluated using the newer defaults: {target, grader} config.yaml schema instead, but confirmed via pickTargetName() in apps/cli/src/commands/eval/targets.ts that it isn't actually wired into eval run's target/grader resolution yet (it falls back to the literal string 'default', not config.yaml's defaults.target) — so adding it here would be inert decoration. Flagging as a separate framework gap.

• --results-repo current/. — the exact syntax documented in --help ("current/. for the source repo") — fell through resultsRepoOverride()'s string checks and got treated as an owner/repo GitHub shorthand, producing git clone https://github.com/current/..git (404). Added current/. as a recognized alias for ./current.

• metadata.eval_file in summary.json could be an absolute host path. Two call sites in run-eval.ts (writeInitialRunSummaryArtifact's stub write, and the final writeArtifactsFromResults write) built evalFile from activeTestFiles[0] without relativizing against cwd, unlike a third nearby call site that already does path.relative(cwd, testFilePath). Applied the same pattern to both.

Test plan

• bun apps/cli/src/cli.ts validate .agentv/targets.yaml — valid (only pre-existing unrelated provider: codex warnings remain).
• bun run typecheck (workspace) — clean.
• bunx biome check on touched files — clean.
• bun test apps/cli/test/eval.integration.test.ts — 30/33 pass; the 3 failures are pre-existing on unmodified main (verified via git stash), unrelated to this change (a different runtime_source.eval_files path issue).
• Live dogfood: bun apps/cli/src/cli.ts eval run examples/features/rubric/evals/operators.eval.yaml --target llm --workers 1 --threshold 0.8 --results-repo current/. --results-branch agentv/results/v1 --results-push against a live Azure target + real llm-rubric grader — 100% pass, --results-repo current/. correctly resolved to the source repo, pushed to agentv/results/v1, and the pushed summary.json's metadata.eval_file is examples/features/rubric/evals/operators.eval.yaml (repo-relative).

🤖 Generated with Claude Code